Skip to content

Board: totals across requests rank only under Throughput; rig pages chart decode vs concurrent requests - #141

Merged
jackwsmth merged 2 commits into
mainfrom
issue-130-streams
Nov 16, 2026
Merged

jackwsmth merged 2 commits into
mainfrom
issue-130-streams

Conversation

@t8

@t8 t8 commented Nov 15, 2026

Copy link
Copy Markdown
Member

Follow-up to #135/#136 (issue #130, step 3). A result measured across several concurrent requests with decodeScope: total is the machine's throughput, not what one request sees, yet on All results (the default tab) it ranked against single-request numbers and set a rig's "best decode tok/s". With throughput submissions now on the site (e.g. 627 tok/s across 144 streams on a 3-NUC rig next to 33 tok/s single-stream), that misrepresents inference speed.

What changes

  • Throughput setup. A new tab ranks results with concurrency > 1 and decodeScope: total against each other. Rules shown: ≥ 2 requests, total across requests. Per-request multi-stream results keep ranking in 4 requests as before.
  • All results keeps per-request numbers only. Totals are excluded from the default tab and from every "best decode tok/s" summary: rig cards, hardware/runtime/custom-build summaries, the home page's top rigs, top results, and the rig page's best-per-model chart. They are still listed in a rig's results and rank on Throughput; a result page says so (rank.setup: 'throughput'). Legacy batchSize results are untouched (the field never had a fixed meaning).
  • Rig page: decode tok/s by concurrent requests. Results of one model, quant, runtime, build and part measured at two or more request counts are drawn as curves, one chart for totals and one for per-request rates (different quantities, so different y-axes). A single-request result seeds both. Points link to their result. Nothing is drawn for a setup measured at only one count; the caption says what to submit to extend a line.
  • Guidance (/guidelines, CONTRIBUTING, results/README.md): record concurrency and decodeScope rather than "set the batch size"; totals rank under Throughput; several request counts of the same setup draw a curve.

Tests

  • scripts/tests/streams.test.mjs (new): seeding of both curves from a single request, totals vs per-request separation, best-of repeats at a count, legacy/hidden/ambiguous exclusions, series keys, and the Throughput matcher.
  • scripts/tests/board-submissions.test.mjs: an 8-request total (500 tok/s) stays off All results and Standard, ranks alone on Throughput, is counted in setupCounts, and never becomes the rig's best decode.
  • npm run typecheck, npm run build, results:test (90), agents:test (25) all pass.

Data

Rig 8's Inkling concurrency survey (15 points, 1–176 streams) recorded its stream count in the legacy batchSize only; I am backfilling concurrency and decodeScope: total on those results through the agent API so the rig page draws the curve once this merges.

…hart decode vs concurrent requests (#130)

- setups: a Throughput setup for results with concurrency > 1 and decodeScope total; All results keeps per-request numbers only
- supabase: totals never count as a rig's, part's, runtime's or build's best decode speed, nor in top results or the rig chart
- lib/streams + StreamsChart: group a rig's results by model, quant, runtime, build and part into total and per-request curves against request count; rig page draws them
- guidance, CONTRIBUTING and results/README: record concurrency and decodeScope instead of batch size; totals rank under Throughput
- tests: series grouping, the Throughput setup, and a total that stays off All results and the best-decode summaries
@vercel

vercel Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
intelinside Ready Ready Preview Nov 16, 2026 9:37am UTC

Request Review

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
No submission files to import.

Site / sign up · Workflow details and retry

The rig page's share line and the hardware page's best-per-runtime chart
still counted results with decodeScope total, so rig 21 shared 'Best 627
tok/s' (144 requests combined).
@jackwsmth
jackwsmth merged commit baa6de1 into main Nov 16, 2026
6 checks passed

This branch was successfully deployed

1 active deployment
Preview — f53c39ca Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants